Summary

Large orchestration payloads are externalized to Azure Blob Storage by the AzureBlobPayloads extension as blob:v1:<container>:<blobName> tokens. The DTS backend stores those tokens but cannot delete the backing blobs — it has no storage credentials; only this SDK can. This PR implements the worker/SDK side of large-payload blob auto-purge.

Design (opt-in singleton durable entity + orchestration job)

Instead of an always-on background stream, this mirrors the existing src/ExportHistory feature: an opt-in, whole-scheduler singleton durable entity + orchestration job that drains soft-deleted payload rows the backend exposes and deletes their blobs, then acks so the backend can hard-delete the rows.

• PayloadStore.DeleteAsync — added as a virtual method (default throws NotSupportedException, so it is non-breaking for existing external subclasses). BlobPayloadStore overrides it to decode the token and call DeleteIfExistsAsync (idempotent — deleting a missing blob is a no-op).
• BlobPurgeJob (TaskEntity singleton) — Create is a no-op when already Active so racing client processes don't disturb the running job (intentionally softer than ExportJob.Create, which throws). Run starts a fixed-instance-id orchestrator.
• BlobPurgeJobOrchestrator (perpetual) — each cycle: fetch a batch of tombstones, delete the blobs with capped parallelism (32), ack only the successful deletions (failed tokens stay tombstoned to retry), idle on a 1-minute timer when there's nothing to purge, and ContinueAsNew every 5 cycles to keep history small. Activities use a small retry policy.
• ExecuteBlobPurgeJobOperationOrchestrator — client → entity bridge (mirrors export).
• Activities (constructor DI): GetTombstonedPayloadsActivity, DeleteExternalBlobActivity (returns false + logs on failure so one bad token can't fail the batch), AckPurgedPayloadsActivity.
• BlobPurgeBackendClient — resolves the worker's intercepted gRPC CallInvoker from the named GrpcDurableTaskWorkerOptions and calls the two new unary RPCs.
• Options — LargePayloadStorageOptions.AutoPurge (opt-in, default false) and PayloadPurgeBatchSize (default 500).
• Enablement — client-side BlobPurgeJobStarter (IHostedService) ensures the singleton job when AutoPurge is enabled, on a background task that does not block host startup and retries until the backend is reachable. The worker always registers the entity/orchestrators/activities (not gated on AutoPurge) so a client-enabled job always has something to execute.

gRPC contract

Two new unary RPCs added to TaskHubSidecarService in src/Grpc/orchestrator_service.proto (worker is the client; wire paths /TaskHubSidecarService/GetTombstonedPayloads and /AckPurgedPayloads):

rpc GetTombstonedPayloads(GetTombstonedPayloadsRequest) returns (GetTombstonedPayloadsResponse);
rpc AckPurgedPayloads(AckPurgedPayloadsRequest) returns (AckPurgedPayloadsResponse);

message TombstonedPayload { int32 partitionId = 1; int64 instanceKey = 2; int64 payloadId = 3; string token = 4; }
message PayloadPurgeAck   { int32 partitionId = 1; int64 instanceKey = 2; int64 payloadId = 3; }
message GetTombstonedPayloadsRequest  { int32 limit = 1; }
message GetTombstonedPayloadsResponse { repeated TombstonedPayload payloads = 1; }
message AckPurgedPayloadsRequest      { repeated PayloadPurgeAck acks = 1; }
message AckPurgedPayloadsResponse     { }

C# stubs are generated at build time by Grpc.Tools (not committed). The authoritative proto change is a follow-up in microsoft/durabletask-protobuf#76; once it merges, src/Grpc/orchestrator_service.proto should be re-synced from upstream (content identical to what's here). The vendored change lets this PR build/test standalone.

Testing

• dotnet build Microsoft.DurableTask.sln — succeeds, 0 errors.
• dotnet test test/AzureBlobPayloads.Tests — 11 passed (BlobPayloadStore delete/idempotency + BlobPurgeJob.Create no-op-when-Active + options defaults).
• dotnet test test/ExportHistory.Tests — 147 passed (shared patterns unaffected).

Notes / deviations

• BlobPurgeJobStatus.Stopped is the 0/default value (mirroring ExportJobStatus.Pending=0) so a freshly initialized entity never appears Active.
• Added a RecordPurged entity op to track a cumulative PurgedCount.

Draft — not for merge until the authoritative proto PR lands and the backend serves the RPCs on TaskHubSidecarService.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com